Method and device for detecting lanes, driver assistance system and vehicle

ABSTRACT

A method of detecting lanes includes the steps: capturing (S1) a camera image (K) of a vehicle environment by a camera device (2) of a vehicle (5); determining (S2) feature points (P1 to P15) in the camera image (K), which feature points correspond to regions of possible lane boundaries (M1, M2); generating (S3) image portions of the captured camera image (K) respectively around the feature points (P1 to P15); analyzing (S4) the image portions using a neural network to classify the feature points (P1 to P15); and determining (S5) lanes in the vehicle environment taking account of the classified feature points (P1 to P15).

FIELD OF THE INVENTION

The invention relates to a method for detecting lanes, a device fordetecting lanes, a driver assistance system, and a vehicle.

BACKGROUND INFORMATION

Driver assistance systems produce a model of the vehicle surroundingsbased on a plurality of sensor data. In addition to detecting obstaclesand further road users, the detection of lanes is particularly relevant.By establishing the vehicle's movement in the lanes, the driver can begiven an early warning in the event of the vehicle unintentionallyleaving a lane. Driver assistance systems can additionally support thedriver with remaining in a particular lane or can steer the vehicleautonomously along the lane.

The lanes are usually detected by establishing the lane boundaries, aterm used in particular to denote lane markings and curbsides. Theprinted publication DE 10 2015 209467 A1 discloses a method ofestimating lanes using feature vectors which are establishedindependently of each other based on various sensors.

Since the mostly white or yellow lane markings stand out from the darkroad, edge detection methods which capture the transition between thelighter and darker areas can be deployed in order to establish the lanemarkings. The use of Sobel filters is widespread, wherein a differencein the brightness values of two neighboring areas is substantiallycalculated. At a constant brightness, this difference is averaged atzero, while values which differ from zero are produced at the edges.

The deployment of neural networks for detecting objects and, inparticular, for detecting lane markings is becoming increasinglywidespread. The advantage of such methods is that different surroundingconditions, for instance variations in the brightness relating to thetime of day or weather, can be taken better account of than is the casefor static Sobel filters.

However, since detected images are typically evaluated by means ofneural networks pixel by pixel, the computational cost is relativelyhigh, resulting in an increased energy consumption and a reducedevaluation speed.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to make possible arapid and precise detection of lanes.

This object can be achieved by a method, a device, a driver assistancesystem, and a vehicle respectively having the features of respectiveaspects of the invention as set forth herein.

According to a first aspect, the present invention accordingly creates amethod for detecting lanes, wherein a camera image of an environment ofa vehicle is captured by means of a camera device of the vehicle.Feature points which correspond to regions of possible lane boundariesare determined. Image portions of the captured camera image aregenerated around each feature point. The image portion is analyzed usinga neural network in order to classify the feature points. Finally, lanesin the vehicle environment are detected, taking account of theclassified feature points.

According to a second aspect, the present invention accordingly createsa device for detecting lanes, which has an interface for receiving acamera image of an environment of the vehicle captured by means of acamera device of a vehicle. The device additionally comprises acomputing apparatus which determines feature points in the camera imagereceived by means of the interface, which feature points correspond toregions of possible lane boundaries. The computing apparatus generatesimage portions of the captured camera image around a feature point,analyzes the image portions using a neural network in order to classifythe feature points and determines lanes in the vehicle environmenttaking account of the classified feature points.

According to a third aspect, the invention relates to a driverassistance system for a vehicle, comprising a camera device whichcaptures a camera image of an environment of the vehicle, and comprisinga device for detecting lanes on the basis of the camera image capturedby the camera device.

According to a fourth aspect, the invention relates to a vehiclecomprising a driver assistance system.

The invention provides a two-stage detection of lanes. During a firststage, the camera image is analyzed by means of preferably traditionalmethods for detecting features and feature points are determined, whichcorrespond to regions of possible lane boundaries. During this firststep, less relevant regions, i.e. those regions in which in allprobability no lane boundaries are to be expected, are alreadyeliminated in a coarse evaluation. This reduces the computational costduring the further processing of the data.

In a subsequent second stage, a more detailed analysis of the discoveredfeature points is performed. To this end, image portions around thefeature points are evaluated by means of a neural network. As a result,a classification of the feature points can be performed, i.e. acategorization of whether the feature point corresponds to a laneboundary or not. The lanes are determined on the basis of those featurepoints which have been classified as feature points corresponding tolane boundaries.

A further advantage of neural networks compared with Sobel filters isthe consideration of larger image portions. Thus, a Sobel filter has ingeneral a capturing range of, for instance, 3×3 pixels, while the imageportion for the neural network can have, for example, a capturing rangeof 128×128 pixels. These larger image portions make it possible for theneural network to capture the context of the surroundings in order toclassify the feature point. Thus, for example, bushes within the imageportion can indicate a guardrail and, therefore, a misidentification oran outlier. Consequently, a surrounding region of the image portion isalso advantageously evaluated.

The invention consequently preferably combines traditional methods withthe use of neural networks. However, due to the pre-filtering, onlyspecific regions of the camera image have to be analyzed by means of theneural network and a total evaluation of all of the pixels can bedispensed with. As a result, the lane detection can be performed quicklyand efficiently. At the same time, the high detection accuracies of theneural networks can be fully utilized.

In order to prevent lane boundaries not being detected, the thresholdswhich have to be exceeded in order to detect a pixel as a feature pointare preferably set relatively low. The resulting false identificationsare subsequently filtered out again by means of the neural network. IfSobel filters are exclusively used, the thresholds are typicallyselected in such a way that as few misidentifications as possible occur.As a result, low-contrast lane boundaries are not detected however.Thanks to the two-stage method according to the invention, no suchcompromise has to be made and the detection accuracy is higher.

According to a preferred further development of the method, the featurepoints are determined using edge detection algorithms. The edgedetection is consequently effected by means of traditional methods andpreferably by using Sobel filters.

According to a preferred further development of the method, the possiblelane boundaries comprise lane markings. In particular, lane markings canbe exclusively detected as possible lane boundaries. Lane markings orroad markings are to be construed to be colored markers on the surfaceof the roads, which divide or delimit the individual lanes. According tofurther embodiments, the lane boundaries can however also comprisecurbsides, guardrails or vegetation, in particular if road markings aremissing.

According to a preferred further development of the method, the neuralnetwork is a convolutional neural network.

According to a preferred further development of the method, the neuralnetwork is taught by means of predefined training data from a database.

According to a further development, the training data compriseillustrations of lane boundaries. The training data can additionallycomprise illustrations of structures which do not constitute lanemarkings. These illustrations are preferably selected in such a way thatthey illustrate structures and objects which can typically be wronglyidentified as lane markings by means of traditional edge detection.Examples can be illustrations of vehicles or of guardrails which do notconstitute a lane boundary. As a result, the neural network is taught insuch a way that illustrations of lane boundaries are distinguished fromillustrations which do not show lane boundaries.

The image portions used for analysis by means of the neural networkpreferably have a predefined number of pixels or a predefined size. Theillustrations selected as training data preferably correspond to typicalillustrations of images of the vehicle environment of said predefinednumber of pixels or size captured by means of a camera device of avehicle.

The illustrations are preferably at least partially generated withdifferent brightnesses. For example, the illustrations can be generatedat various times of day. The illustrations can also be generated undervarious lighting conditions, for instance on poorly lit or well-litstreets. The illustrations can additionally be captured in variousweather conditions, for instance sunshine, fog, rain or snow.

According to a preferred further development of the method, the coursesof lane boundaries are determined in that neighboring feature pointswhich have been classified as belonging to a lane boundary areinterpolated. On the basis of the courses of the lane boundaries, thelanes which are usable by the vehicle are established.

According to a preferred further development of the device, thecomputing apparatus determines the feature points using edge detectionalgorithms.

According to a preferred further development of the device, thecomputing apparatus uses Sobel filters in order to detect edges.

According to a preferred further development, the computing apparatus ofthe device is configured to determine courses of lane boundaries byinterpolating neighboring feature points which have been classifiedaccordingly as a lane boundary. The computing apparatus establishes thelanes which are usable by the vehicle on the basis of the courses of thelane boundaries.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is explained in greater detail below withreference to the embodiment examples indicated in the schematic figuresof the drawings.

Therein:

FIG. 1 shows a schematic block diagram of a device for detecting lanesaccording to an embodiment of the invention;

FIG. 2 shows a schematic camera image captured by a camera device;

FIG. 3 shows illustrations as training data for a neural network;

FIG. 4 shows lane boundaries detected in a camera image;

FIG. 5 shows a schematic block diagram of a driver assistance system;

FIG. 6 shows a schematic block diagram of a vehicle; and

FIG. 7 shows a flow chart of a method for detecting lanes.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS OF THE INVENTION

Further possible configurations, further developments andimplementations of the invention also comprise combinations of featuresof the invention described above or below with respect to the embodimentexamples, which are not explicitly indicated.

The appended drawings are intended to convey a further understanding ofthe embodiments of the invention. They illustrate embodiments and, inconnection with the description, serve to explain the principles andconcepts of the invention. Other embodiments and many of the indicatedadvantages are set out with respect to the drawings. The same referencenumerals indicate the same or similarly acting components.

FIG. 1 shows a schematic block diagram of a device 1 for detectinglanes.

The device 1 comprises an interface 11 which is configured to receiveand output data wirelessly or via a wired connection. In particular, theinterface 11 receives camera data and transfers said data to a computingapparatus 12 of the device 1. The camera data comprise at least onecamera image which has been generated by a camera device 2 of a vehicle.The camera image can also be combined from multiple individual images ofa vehicle camera of the camera device 2 or from multiple images of aplurality of vehicle cameras of the camera device 2.

The computing apparatus 12 analyzes the camera image by means of imagedetection methods in order to extract feature points which correspond toregions having lane boundaries in the vehicle environment. The computingapparatus 12 comprises at least one microprocessor in order to performthe calculation steps.

The computing apparatus 12 generates a respective image portion aroundeach of the feature points. This image portion serves as an inputvariable for a neural network which assesses the image portion. Theneural network is preferably a convolutional neural network. Aprobability with which the image portion illustrates a lane boundary iscalculated by means of the neural network. If the probability exceeds apredefined threshold, the computing apparatus 12 detects that thefeature point of the image portion corresponds to a lane boundary.

The feature points classified in such away are further evaluated by thecomputing apparatus 12, in order to determine lanes in the environmentof the vehicle. Thus, the computing apparatus 12 can determine thecourse of the lane boundaries in the camera image by interpolatingfeature points neighboring each other, which correspond to laneboundaries. The regions running between lane boundaries can beidentified as lanes, and the computing apparatus 12 can generate anenvironment model.

Individual aspects of the device 1 are depicted more precisely below onthe basis of FIGS. 2 to 4.

Thus, FIG. 2 shows a camera image K captured by a camera device 2. Theillustrated objects comprise a right lane marking 51, a middle lanemarking 52, a guardrail 53 located at the edge of the right lane and avehicle 54 driving on a parallel lane.

The computing apparatus 12 analyzes the pixels of the camera image K bymeans of traditional edge detection methods. In particular, thecomputing apparatus 12 can apply a Sobel filter to each pixel in orderto detect an edge on or in the surroundings of the pixel. The Sobelfilter can take account of 3×3 pixels in the surroundings of the pixelto be examined, but it can also allow for a larger surrounding region ofthe pixel.

The computing apparatus 12 can establish for each pixel whether thepixel is located at or in the proximity of an edge. In particular, thecomputing apparatus 12 can compare the value calculated by means of theSobel filter with a predefined threshold. If the threshold is exceeded,the computing apparatus 12 establishes that the pixel is a feature pointwhich corresponds to a possible lane boundary.

In the camera image K shown in FIG. 2, the computing apparatus 12determines a total of 15 feature points P1 to P15. This is only to beunderstood by way of example. In general, a larger number of featurepoints is generated.

The computing apparatus 12 generates an image portion B1 to B3 for eachfeature point P1 to P15. For the sake of simplicity, only the imageportions for the first three feature points P1 to P3 are marked in FIG.2. The image portions B1 to B3 can have a predefined size of, forexample, 128×128 pixels. The feature point P1 to P15 is, in each case,preferably arranged in the center of the respective image portion B1 toB3.

The generation of the neural network used to further analyze the imageportions B1 to B3 is explained in greater detail in FIG. 3. Accordingly,a database DB with training data is first produced. The training datacomprise illustrations which have been captured by means of a vehiclecamera. These are preferably manually classified into two groups. Afirst group Ta1 to Ta4 comprises illustrations which show laneboundaries. The illustrations can exclusively comprise images of lanemarkings. However, according to further embodiments, the illustrationscan also illustrate curbsides or further lane boundaries. To this end,the illustrations can be produced with various brightnesses or weatherconditions. The second group comprises illustrations Tb1 to Tb4 whichshow objects with edges which are not, however, lane boundaries. Thesecan be illustrations of vehicles Tb1, Tb3, guardrails Tb2 or bridgesTb4.

The neural network is then trained in such a way that the illustrationsof the first group Ta1 to Ta4 are classified as illustrations of laneboundaries, while the illustrations of the second group Tb1 to Tb4 areclassified as illustrations which do not show lane boundaries. Followingthe training phase, the computing apparatus 12 can classify any imageportions B1 to B3 by means of the neural network. To this end, aprobability that the image portion B1 to B3 is an illustration of a laneboundary can first be output by means of the neural network. If thecalculated probability exceeds a predefined threshold, for example 0.5,the computing apparatus 12 classifies the image portion B1 to B3 ascorresponding to a lane boundary.

For the feature points P1 to P15 of the camera image K, the computingapparatus 12 detects, for example, that the feature points P2, P5 to P8of the middle lane marking 52 and the feature points P1, P9 to P13 ofthe right lane marking 51 are feature points which correspond to laneboundaries. Conversely, the measuring points P4, P3, P14, P15 of theguardrail 53 and of the vehicle 54 are discarded as misidentificationssince the illustrated objects are not lane boundaries.

In order to determine the lanes, the computing apparatus 12 preferablymerely includes those feature points P1 to P15 which have been detectedas corresponding to lane boundaries.

The computing apparatus 12 can then determine the corresponding laneboundaries by interpolating the neighboring remaining feature points P1to P15 or pixels.

As shown in FIG. 4, the computing apparatus 12 detects, for example, afirst lane boundary M1 and a second lane boundary M2 for the cameraimage K illustrated in FIG. 2. The computing apparatus 12 accordinglydetermines that a lane F runs between the lane boundaries M1, M2.

The described detection of the lanes is preferably performediteratively, wherein the lane boundaries and lanes already detected areupdated.

A block diagram of a driver assistance system 4 for a vehicle accordingto an embodiment of the invention is depicted in FIG. 5. The driverassistance system 4 comprises a camera device 2 which has one or aplurality of vehicle cameras which are arranged or arrangeable on thevehicle.

The driver assistance system 4 additionally comprises a device 1 fordetecting lanes, which is described above. The device comprises aninterface 12 which is described above and which receives the cameraimages captured by the camera device 2, as well as a computing apparatus12 which determines lanes F on the basis of the camera images.

The driver assistance system 4 can comprise a control device 3 which cancontrol specific driving functions of the vehicle. Thus, the controldevice 3 can control the vehicle as a function of the detected lanes insuch a way that the vehicle is accelerated, braked or steered. Thedriver assistance system 4 can, as a result, make possible asemi-autonomous or autonomous control of the vehicle. The control device3 can additionally be configured to output a warning signal if thevehicle leaves the detected lane F in order to warn the driver againstan unintentional departure from the lane F.

A block diagram of a vehicle 5 according to an embodiment of theinvention is depicted in FIG. 6. The vehicle 5 can, for instance, be acar, a truck or a motorcycle. The vehicle 5 comprises a driverassistance system 4, which is described above, comprising a device 1 fordetecting lanes F in the surroundings of the vehicle 5.

FIG. 7 shows a flow chart of a method for detecting lanes F according toan embodiment of the invention.

In a method step S1, a camera image of a vehicle environment is capturedby means of a camera device 2. To this end, multiple individual imagescan also be combined.

In the further method step S2, the individual pixels of the camera imageare evaluated by means of an edge detection method, in order todetermine feature points P1 to P15. To this end, Sobel filters can forexample be used in order to detect edges. If the values calculated bymeans of the Sobel filter exceed a predefined threshold, the pixels areidentified as feature points P1 to P15 which can correspond to laneboundaries.

An image portion B1 to B3 is generated around each feature point P1 toP15 in a method step S3. The feature point P1 to P15 can preferably belocated in a center of a square image portion B1 to B3. The size of theimage portion can, for example, be 128×128 pixels. However, theinvention is not restricted to this. Thus, the image portion does notnecessarily have to have a square or rectangular configuration. The formof the image portion can be selected, for example, as a function of theperspective representation of the camera device 2.

In a method step S4, the image portions are analyzed using a neuralnetwork. To this end, the neural network is produced or taught on thebasis of training data from a database. The training data comprise theillustrations of lane boundaries or of lane surroundings without laneboundaries described in connection with FIG. 3. Following the trainingphase, the neural network is configured to analyze and classify anyimage portions. It is detected for each image portion whether theillustrated region of the vehicle environment illustrates a laneboundary or not. The feature points are classified accordingly. Thosefeature points which are classified accordingly as a lane boundary bymeans of the neural network are further evaluated, while the remainingfeature points are discarded.

On the basis of the remaining feature points, the courses of laneboundaries M1, M2 are determined in a method step S5. On the basis ofthe courses of the lane boundaries M1, M2, lanes F which are usable bythe vehicle 5 are detected.

On the basis of the detected lanes F, warning signals can additionallybe output or a semi-autonomous or autonomous control of the vehicle 5can be performed.

REFERENCE NUMERALS

-   1 Device for detecting lanes-   2 Camera device-   3 Control device-   4 Driver assistance system-   5 Vehicle-   11 Interface-   12 Computing apparatus-   51 Right lane marking-   52 Middle lane marking-   53 Guardrail-   54 Vehicle-   F Lane-   M1, M2 Lane boundaries-   P1 to P15 Feature points-   E1 to B3 Image portions-   Ta1 to Ta4 Illustrations of lane boundaries-   Tb1 to Tb4 Illustrations of objects which are not lane boundaries

The invention claimed is:
 1. A method of detecting a lane on a roadway, comprising the steps: with a camera of a vehicle, capturing a camera image of a vehicle environment including a roadway on which the vehicle is driving; determining feature points in the camera image by analyzing the camera image with image processing that does not use a neural network, wherein the feature points correspond to points on possible lane boundaries of at least one lane on the roadway; selecting a respective image portion of the camera image having a respective predefined size around each respective one of the feature points; analyzing the image portions using a neural network to classify the feature points thereof regarding whether or not the feature points represent actual lane boundaries; and determining the at least one lane on the roadway in the vehicle environment based on the feature points that have been classified as representing the actual lane boundaries.
 2. The method according to claim 1, further comprising operating a driver assistance system of the vehicle in response to and dependent on the at least one lane that has been determined, to autonomously or semi-autonomously control a driving operation of the vehicle with respect to guidance thereof relative to a respective lane of the at least one lane that has been determined.
 3. The method according to claim 1, further comprising operating a driver assistance system of the vehicle in response to and dependent on the at least one lane that has been determined, to output a warning signal to a driver of the vehicle when the vehicle leaves a respective lane of the at least one lane that has been determined.
 4. The method according to claim 1, wherein the respective predefined size of the respective image portion is defined based on a predefined total number of pixels of the respective image portion, predefined pixel dimensions of the respective image portion, or a predefined physical size of the respective image portion.
 5. The method according to claim 1, wherein the feature points are respective feature point pixels in the camera image, and the selecting of the respective image portion comprises defining the respective image portion having the predefined size around a respective one of the feature point pixels so that the respective feature point pixel is located within the respective image portion.
 6. The method according to claim 1, wherein the feature points are respective feature point pixels in the camera image, and the selecting of the respective image portion comprises defining the respective image portion having the predefined size around a respective one of the feature point pixels so that the respective feature point pixel is located at a center of the respective image portion.
 7. The method according to claim 1, wherein the image processing for the determining of the feature points uses edge detection algorithms.
 8. The method according to claim 7, wherein the edge detection algorithms use Sobel filters.
 9. The method according to claim 1, wherein the possible lane boundaries comprise lane markings.
 10. The method according to claim 1, wherein the neural network is a convolutional neural network.
 11. The method according to claim 1, further comprising teaching the neural network using predefined training data from a database.
 12. The method according to claim 11, wherein the training data comprise illustrations of example lane boundaries and illustrations of example non-boundary structures that do not constitute lane boundaries.
 13. The method according to claim 12, wherein the illustrations have been generated at least partially with different brightnesses and/or different times of day and/or different weather conditions.
 14. The method according to claim 1, further comprising determining courses of the actual lane boundaries by interpolating between neighboring ones of the feature points that have been classified as representing the actual lane boundaries, and wherein the determining of the at least one lane is further based on the courses of the actual lane boundaries.
 15. A device for detecting a lane on a roadway, comprising: an interface configured to receive, from a camera device of a vehicle, a camera image of a vehicle environment including a roadway on which the vehicle is driving; and a computing apparatus which is configured: to determine feature points in the camera image by analyzing the camera image with image processing that does not use a neural network, wherein the feature points correspond to points on possible lane boundaries of at least one lane on the roadway; to select a respective image portion of the camera image having a respective predefined size around each respective one of the feature points; to analyze the image portions using a neural network to classify the feature points thereof regarding whether or not the feature points represent actual lane boundaries; and to determine the at least one lane on the roadway in the vehicle environment based on the feature points that have been classified as representing the actual lane boundaries.
 16. The device according to claim 15, wherein the computing apparatus is configured to perform the image processing for determining the feature points by using edge detection algorithms.
 17. The device according to claim 16, wherein the computing apparatus is configured to use Sobel filters for the edge detection algorighms.
 18. The device according to claim 15, wherein the computing apparatus is further configured to determine courses of the actual lane boundaries by interpolating between neighboring ones of the feature points that have been classified as representing the actual lane boundaries, and to determine the at least one lane further based on the courses of the actual lane boundaries.
 19. A driver assistance system for a vehicle, comprising: a device for detecting a lane on a roadway according to claim 15, and the camera device configured to capture the camera image of the vehicle environment of the vehicle.
 20. A vehicle comprising the driver assistance system according to claim 19 and a vehicle body. 